System and Method for Providing Additional Information Using Image Matching

ABSTRACT

The present invention relates to a system and method for providing additional information using image matching. The present invention provides a system for providing additional information using image matching, including: a image input unit configured to receive a video or an image input from a user terminal; a query image determination unit configured to determine a query image, i.e., a query target, from the video or image received from the image input unit; a feature point extraction unit configured to extract feature points of the query image determined by the query image determination unit, and to generate feature point information of the extracted feature points; a reference meta-information database configured to store a structure image identifier (ID) of a structure image of a structure on which an original image has been installed, structure image feature point information of the structure image, and additional information to be provided to the user terminal in association with the structure image ID; an image matching unit configured to determine a matching structure image ID by comparing the feature point information of the query image generated by the feature point extraction unit with the structure image feature point information stored in the reference meta-information database; and an additional information provision unit configured to provide the additional information stored in association with the structure image ID determined by the image matching unit to the user terminal, and a method using the same.

CROSS REFERENCE TO RELATED APPLICATIONS

This patent application is a continuation of U.S. patent applicationSer. No. 14/441,310, filed May 7, 2015, which is a U.S. national stageentry of PCT Patent Application No. PCT/KR2013/001709, filed Mar. 4,2013, which claims priority to Korean Patent Application No.10-2013-0019253, filed Feb. 22, 2013 and to Korean Patent ApplicationNo. 10-2012-0130365, filed Nov. 16, 2012.

TECHNICAL FIELD

The present invention relates to a system and method for providingadditional information using image matching and, more particularly, to asystem and method that can determine the location information of a userterminal based on a video or an image input from the user terminal usingimage matching technology and then provide various types of additionalinformation based on the determined location information.

BACKGROUND ART

In a daily life environment, there are an enormous number of imageseverywhere you look around. The majority of these images are intended totransfer specific messages to viewers. For example, with respect to amethod of performing advertisement using an image, although anadvertisement has been provided via an outdoor billboard to which anoutput very-large image has been attached in its early stage, thismethod is recently being changed to a method of transferring a digitizedimage or video to the public present in a specific time span and in aspecific space via a large-sized monitor. However, most current methodsof transferring a message via an image still remain in a unilateral formin which a message is transferred from an image from a user.

In this respect, there is a need for the development of technology forenabling a provider who provides a specific image or video, such as anadvertisement, to automatically determine whether a user has observedthe corresponding image or video and then provide appropriate additionalinformation to the user accordingly.

Meanwhile, Korean Patent Application Publication No. 10-2013-0008109(published on Jan. 22, 2013) relates to “a mobile terminal, a newsprovision apparatus, and a location-based news provision method,” anddiscloses a method of receiving location information from a mobileterminal and transmitting related news information based on the receivedlocation information to the corresponding terminal. However, the methodrequires that location information is accurately received from theterminal first, and thus the terminal itself should acquire locationinformation. Accordingly, the method has a limitation in that newsinformation cannot be provided to a terminal that cannot acquirelocation information because the terminal has no location informationacquisition means or a terminal that does not transmit locationinformation.

DISCLOSURE Technical Problem

Accordingly, the present invention has been contrived keeping in mindthe above problems occurring in the prior art, and an object of thepresent invention is to provide a system and method that canautomatically determine whether a user has observed a structure image orvideo, such as an advertisement, attached to a structure, such as anoutdoor billboard, and that can provide an appropriate type ofadditional information to the user accordingly.

Another object of the present invention is to provide a system andmethod that enable a query image, i.e., a comparison target, to be moreaccurately determined from among images or videos transmitted from auser, thereby increasing the efficiency of image matching.

A further object of the present invention is to provide a system andmethod that make use of the structure-related surrounding information ofthe feature point information of a structure image upon image matching,thereby enabling information about the identity and location of thestructure image transmitted by a user to be accurately determined.

Yet another object of the present invention is to provide a system andmethod that can determine information about an angle at which a userterminal is located based on a structure image and information about adirection in which the user terminal is moving and that can provideadditional information based on the determined information, therebyincreasing the efficiency and accuracy of information provision.

Technical Solution

In order to accomplish the above objects, the present invention providesa system for providing additional information using image matching,including: a image input unit configured to receive a video or an imageinput from a user terminal; a query image determination unit configuredto determine a query image, i.e., a query target, from the video orimage received from the image input unit; a feature point extractionunit configured to extract the feature points of the query imagedetermined by the query image determination unit, and to generate thefeature point information of the extracted feature points; a referencemeta-information database configured to store the structure imageidentifier (ID) of a structure image of a structure on which an originalimage has been installed, the structure image feature point informationof the structure image, and additional information to be provided to theuser terminal in association with the structure image ID; an imagematching unit configured to determine a matching structure image ID bycomparing the feature point information of the query image generated bythe feature point extraction unit with the structure image feature pointinformation stored in the reference meta-information database; and anadditional information provision unit configured to provide theadditional information stored in association with the structure image IDdetermined by the image matching unit to the user terminal.

In this case, the query image determination unit may include: a frameextraction unit configured to extract the input video as a plurality offrame images when the video is input from the user terminal; and a queryimage selection unit configured to determine the query image, i.e., thequery target, from among the frame images extracted by the frameextraction unit.

The query image selection unit may determine the query image based onthe gradient values of the frame images.

The query image may be determined based on scores that are calculated bythe following equation:

$\begin{matrix}{{score} = \frac{\int{{{\nabla{I\left( {x,y} \right)}}}^{2}{x}{y}}}{\int{{{I\left( {x,y} \right)}}^{2}{x}{y}}}} \\{= \frac{{\sum\limits_{y}{\sum\limits_{x}\left( {{I\left( {{x + 1},y} \right)} - {I\left( {{x - 1},y} \right)}} \right)^{2}}} + \left( {{I\left( {x,{y + 1}} \right)} - {I\left( {x,{y - 1}} \right)}} \right)^{2}}{\sum\limits_{y}{\sum\limits_{x}{I\left( {x,y} \right)}^{2}}}}\end{matrix}\quad$

where (x, y) is x-axis and y-axis coordinate values of an frame image,and I(x, y) is an image data value at the coordinates (x, y) of theframe image.

The reference meta-information database may further include the originalimage ID of the original image and the original image feature pointinformation of the original image; and the structure image feature pointinformation may include feature point information inside an originalimage region and feature point information outside the original imageregion.

The feature point information outside the original image region may beformed by feature point information that does not match the originalimage feature point information when matching between the original imagefeature point information and the structure image feature pointinformation is performed.

The image matching unit may determine N (where N is a natural number)structure image IDs of highest matching scores by comparing the featurepoint information of the query image extracted by the feature pointextraction unit with the structure image feature point informationstored in the reference meta-information database, and may finallydetermine a structure image ID by comparing the feature pointinformation outside original image region corresponding to thedetermined structure image IDs with the feature point information of thequery image.

The image input unit may receive the location information of the userterminal from the user terminal along with the video or image; thereference meta-information database may include structure locationinformation stored in association with the structure image ID of thestructure on which the original image has been installed; and the imagematching unit may determine N (where N is a natural number) structureimage IDs of highest matching scores by comparing the feature pointinformation of the query image extracted by the feature point extractionunit with the structure image feature point information stored in thereference meta-information database, and may finally determine astructure image ID by comparing feature point information outside theoriginal image region corresponding to the determined structure imageIDs with the feature point information of the query image.

The system may further include a direction calculation unit configuredto determine the first directional angle information of the userterminal directed toward the structure image.

The image input unit may additionally receive a video or an image inputfrom the user terminal within a specific time interval; the query imagedetermination unit, the feature point extraction unit and the imagematching unit may determine a query image, i.e., a query target, fromthe additionally received video or image, extract feature points andthen determine a matching structure image ID; the direction calculationunit may determine the second directional angle information of the userterminal directed toward the structure image and determine the directionof movement of the user terminal based on the first directional angleinformation and the second directional angle information; and theadditional information provision unit may provide the stored additionalinformation to the user terminal based on the structure image IDdetermined by the image matching unit and the direction of movement ofthe determined user terminal.

In accordance with another aspect of the present invention, there isprovided a method of providing additional information using imagematching, including: a first step of receiving a video or an image inputfrom a user terminal; a second step of determining a query image, i.e.,a query target, from the received video or image; a third step ofextracting the feature points of the determined query image andgenerating the feature point information of the extracted featurepoints; a fourth step of determining a matching structure image ID bycomparing the feature point information of the feature points of theextracted query image with structure image feature point information;and a fifth step of providing additional information, stored inassociation with the determined structure image ID, to the userterminal.

Advantageous Effects

In accordance with the present invention, there are provided a systemand method that can automatically determine whether a user has observeda structure image or video, such as an advertisement, attached to astructure, such as an outdoor billboard, and that can provide anappropriate type of additional information to the user accordingly.

Furthermore, the present invention has the effect of providing a systemand method that enable a query image, i.e., a comparison target, to bemore accurately determined from among images or videos transmitted froma user, thereby increasing the efficiency of image matching.

Furthermore, the present invention has the effect of providing a systemand method that make use of the structure-related surroundinginformation of the feature point information of a structure image uponimage matching, thereby enabling information about the identity andlocation of the structure image transmitted by a user to be accuratelydetermined.

Moreover, the present invention has the effect of providing a system andmethod that can determine information about an angle at which a userterminal is located based on a structure image and information about adirection in which the user terminal is moving and that can provideadditional information based on the determined information, therebyincreasing the efficiency and accuracy of information provision.

DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating the overall configuration andconnection relationship of a system (100) for providing additionalinformation using image matching according to the present invention;

FIG. 2 is a diagram illustrating the configuration of an embodiment ofthe system (100) for providing additional information according to thepresent invention;

FIG. 3 is a diagram illustrating the internal configuration of a queryimage determination unit (120);

FIG. 4 is a diagram illustrating a process in which a specific frame isselected by the query image determination unit (120) using a scorefunction;

FIG. 5 is a diagram illustrating an example of the configuration of areference meta-information database (160);

FIG. 6 is a diagram illustrating feature point information inside anoriginal image region and feature point information outside an originalimage region;

FIG. 7 is a diagram illustrating the operation of an image matching unit(140);

FIG. 8 is a diagram illustrating a process in which a directioncalculation unit determines the direction of movement of a user terminal(200); and

FIG. 9 is a flowchart illustrating an embodiment of a method ofproviding additional information to the user terminal (200) using theadditional information provision system (100) of the present inventiondescribed with reference to FIGS. 1 to 8.

BEST MODE

Embodiments according to the present invention are described in detailbelow with reference to the accompanying diagrams.

FIG. 1 is a diagram illustrating the overall configuration andconnection relationship of a system 100 for providing additionalinformation using image matching according to the present invention.

Referring to FIG. 1, the system 100 for providing additional informationusing image matching (hereinafter simply referred to as the “additionalinformation provision system”) according to the present invention isconnected to a user terminal 200, and performs the function of receivinga video or an image, transmitted from the user terminal 200, from theuser terminal 200, identifying a structure on which the video or imagetransmitted from user terminal 200 has been installed based on thereception, and providing various types of additional information basedon the identification.

The video or image transmitted from the user terminal 200 is a video oran image 400 (hereinafter referred to as the “captured image”) that isacquired by being captured by the user terminal 200. For this purpose,the user terminal 200 is provided with a video acquisition means, suchas a camera or the like, that is known from conventional art.

The captured image 400 captured/acquired by the user terminal 200 and tobe transmitted to the additional information provision system 100 is avideo or an image that is captured in such a way that a user manipulatesthe user terminal 200 while centering on an image or a video installedon a structure (a structure attached image 300).

In this case, the term “structure” refers to a construction on (to)which an image or a video has been installed (attached), and may be, forexample, an outdoor billboard in a street, as illustrated in FIG. 1. Theoutdoor billboard may include an image billboard to which an image hasbeen attached, a video billboard, such as a large-sized display means,etc. For example, an automobile-shaped advertising image (the attachedimage 300) is attached to such a structure, as illustrated in FIG. 1.The user terminal 200 performs capturing while centering on thestructure attached image 300, and transmits the captured image 400 tothe additional information provision system 100.

Meanwhile, the structure attached image 300 may be a poster or a movieposter apart from an advertising image or an advertising video. Althoughthe term “structure attached image 300” has been used in connection withFIG. 1 for ease of description, this is defined as corresponding to aconcept including not only an image (that is, a still image) but also amoving image (a video), and is used based on the same concept in thefollowing description.

Although the user terminal 200 performs capturing while centering on thestructure attached image 300 installed on a structure through themanipulation of a user, the actually captured image 400 may include partor all of the structure attached image 300 and thus may include an imageor a video of the structure. Accordingly, the video or image (capturedimage 400) transmitted to the additional information provision system100 may include not only the structure attached image 300 but also avideo or an image of a structure around the image 300, as indicated bydotted lines in FIG. 1. According to the present invention, when thevideo or image of the structure including the structure attached image300 is included in the video or image transmitted from the terminal 200,image matching may be performed based on the video or image of thestructure including the structure attached image 300, thereby increasingthe accuracy of matching, as will be described later.

It will be apparent that the video or image acquired by the userterminal 200 may not include the video or image of the structure but mayinclude part or all of the structure attached image 300, in which casematching is performed through image matching performed on the structureattached image 300.

The term “structure attached image 300” used herein refers to an imageor a video attached to a structure, and the term “captured image 400”used herein refers to a video or an image that the user terminal 200captures while centering on the structure attached image 300 and that istransmitted to additional information provision system 100. The capturedimage 400 includes part or all of the structure attached image 300, andis defined as a video or an image that is captured to include not onlythe structure attached image 300 but also a video or an image of part orall of a structure.

Meanwhile, the user terminal 200 includes a well-known data transmissionmeans for transmitting the captured image 400 to the additionalinformation provision system 100, such as a communication module fortransmitting data over a mobile communication network, or a wireless LANinterface for performing wireless LAN connection.

The user terminal 200 is based on a concept including, for example, amobile communication terminal, a tablet computer, etc. It is preferredthat the user terminal 200 according to the present invention is amobile communication terminal, such as a smart phone. In anotherexample, the user terminal 200 may be a digital camera including a datatransmission means.

Next, the internal configuration and detailed operation of theadditional information provision system 100 will be described withreference to FIG. 2.

FIG. 2 is a diagram illustrating the configuration of an embodiment ofthe additional information provision system 100 according to the presentinvention.

Referring to FIG. 2, the additional information provision system 100 ofthe present embodiment includes a image input unit 110, a query imagedetermination unit 120, a feature point extraction unit 130, an imagematching unit 140, an additional information provision unit 150, and areference meta-information database 160.

The image input unit 110 performs the function of receiving the video orimage (the captured image 400) transmitted from the user terminal 200and transmitting the received video or image to the query imagedetermination unit 120.

As illustrated in FIG. 1, the user terminal 200 acquires the structureattached image 300 or the captured image 400, i.e., an image or a videoof a structure including the structure attached image 300, and thentransmits the acquired image to the additional information provisionsystem 100. In this case, the user terminal 200 transmits the acquiredimage to the image input unit 110 of the additional informationprovision system 100. In this case, when acquiring the captured image400, the user terminal 200 may capture the image in the form of a video(a moving image) or an image (a still image). The user terminal 200transmits the video or image to the additional information provisionsystem 100. In this specification, the term “video” refers to a movingimage, and the term “image” refers to a still image.

The query image determination unit 120 performs the function ofdetermining a query image, i.e., a query target, from the video or image(the captured image 400) received from the image input unit 110. Theterm “query image” refers to an image that is a target that is comparedwith structure image feature point information stored in the referencemeta-information database 160, which will be described later.

As described above, the user terminal 200 acquires a video or imagewhile centering on the structure attached image 300. In the process ofacquiring a video or an image, there are many cases where a video or animage is acquired while part of the body of a user is shaking or a useris moving. Accordingly, there are many cases where a phenomenon in whichthe captured image 400 acquired by the user terminal 200 and transmittedto the additional information provision system 100 is partially orcompletely blurred occurs. Since a video or an image in which such ablurring phenomenon has occurred acts as one of the principal factorsthat reduce accuracy upon performing image matching (which will bedescribed later), the query image determination unit 120 performs thefunction of determining a query image by selecting a frame of relativelylow image quality degradation, i.e., the clearest frame, from thecaptured image 400, i.e., the video or image transmitted from the userterminal 200, thereby being able to increase the accuracy of imagematching.

FIG. 3 is a diagram illustrating the internal configuration of the queryimage determination unit 120, and illustrates a configuration fordetermining a query image when a video is input from the user terminal200.

Referring to FIG. 3, the query image determination unit 120 includes aframe extraction unit 121 and a query image selection unit 122.

The frame extraction unit 121 performs the function of extracting aninput video as a plurality of frame images when the video is input fromthe user terminal 200. As described above, the term “video” used hereinrefers to a “moving image” (for example, in an avi or mpg file format).Such a video includes a plurality of frame images (i.e., a plurality offrame still images, as is well known. The frame extraction unit 121performs the function of extracting a video as a plurality of frameimages. The number of frames may be set to an appropriate value, forexample, a few tens of frames per second, as required.

The query image selection unit 122 performs the function of selecting aquery image, i.e., a query target, from among frame images extractedframe the extraction unit 121.

Although various well-known conventional methods may be used as a methodof selecting a query image, the present invention may employ thefollowing method. That is, a query image may be selected based on thegradient values of respective frame images.

Gradient is variation in the image data values of respective pixelsconstituting an image. As the value of gradient increases, the edgebecomes clearer and shaking becomes lower. Accordingly, since a frameimage of high gradient can be considered to be a clear image, it ispreferable to select a frame image of high gradient as the query image.

In order to measure the degree of clarity of an image used to select aframe image of high gradient as the query image, a score function isdefined as Equation 1 below:

$\begin{matrix}{{score} = \frac{\int{{{\nabla{I\left( {x,y} \right)}}}^{2}{x}{y}}}{\int{{{I\left( {x,y} \right)}}^{2}{x}{y}}}} \\{= \frac{{\sum\limits_{y}{\sum\limits_{x}\left( {{I\left( {{x + 1},y} \right)} - {I\left( {{x - 1},y} \right)}} \right)^{2}}} + \left( {{I\left( {x,{y + 1}} \right)} - {I\left( {x,{y - 1}} \right)}} \right)^{2}}{\sum\limits_{y}{\sum\limits_{x}{I\left( {x,y} \right)}^{2}}}}\end{matrix}\quad$

where (x, y) is the x-axis and y-axis coordinate values of an frameimage, and I(x, y) is an image data value at the coordinates (x, y) ofthe frame image.

Meanwhile, in Equation 1, the image data value I(x, y) may be defined asthe intensity of a pixel at the coordinates (x, y) of the frame image.The intensity may use, for example, R, G and B values or C, M, Y and Kvalues at each pixel. Alternatively, for example, a gray image obtainedby calculating the average value (R+G+B/3) of the R, G and B values ofeach pixel may be generated, and the value of each gray image rangingfrom 0 to 255 may be defined and used as the intensity of the pixel.That is, in Equation 1, I(x, y) is a value that may be appropriatelyselected based on an image data value, i.e., one of the various pixelvalues that each pixel has.

When the scores of the respective frame images are obtained usingEquation 1, a frame image of the highest score can be selected fromamong the frame images. The selected frame image is an image of thehighest gradient and the image of the highest gradient may be viewed asthe clearest image, and thus this frame image is selected as the queryimage. If required, two or more frame images may be selected as thequery image.

FIG. 4 illustrates a process in which the query image determination unit120 selects a query image from among frame images using the scorefunction of Equation 1. As illustrated in FIG. 4, it can be seen that aframe image having a value that is the highest of scores, i.e., theresult values of Equation 1, is selected as the query image by the queryimage determination unit 120.

Meanwhile, the query image determination unit 120 selects a single inputimage as the query image when the single image other than a video isinput from the user terminal 100. When two or more images are input fromthe user terminal 100, these images may be considered to be respectiveframe images and then the query image may be selected using theabove-described process.

Next, the feature point extraction unit 130 is described with referenceto FIG. 2 again.

The feature point extraction unit 130 performs the function ofextracting the feature points of the query image determined by the queryimage determination unit 120 using the above-described method and thengenerating the feature point information (feature point data) of theextracted feature points.

As is well known, the term “feature point” or “interest point” refers toa point capable of representing the feature of a given image in thegiven image, and refers to a point or a set of points capable ofdesirably describing the feature of an image regardless of variation inthe image, such as the scale, rotation, distortion, etc. of the image.Although the number of feature points may varies with the size andcontent of a given image and the type of method of extracting featurepoints, hundreds to tens of thousands of feature points may be extractedfrom a single photo as an example. These feature points are being widelyused in the fields of image processing and computer vision. For example,feature points are extracted, and corresponding portions are searchedfor in two images using the feature point information (feature pointdata) of the extracted feature points and then used for various tasks,such as object identification, operation tracking, and the determinationof identity between images.

Various methods have been proposed as conventional technology forextracting feature points from a given image and generating featurepoint information. For example, a method using maximum and minimumvalues in the scale space of a Laplacian of Gaussian (LoG) filter or aDifference of Gaussians (DoG) filter is known. Additionally, a method ofextracting feature points using the determinant of a Hessian matrix andthen generating feature point information has been also proposed.

Furthermore, methods, such as the Scale-Invariant Feature Transform(SIFT) algorithm disclosed in U.S. Pat. No. 6,711,293 to David G. Loweand the Speed Up Robust Features (SURF) algorithm disclosed in the paperby H. Bay, A. Ess, T. Tuytelaars, L. V. Gool, “Speeded-Up Robust Feature(SURF),” Journal of Computer Vision and Image Understanding, 110, pp.346-359, 2008, have been proposed as a method of extracting the featurepoints of an image and generating the feature point information of theextracted feature points.

Furthermore, Korean Patent No. 10-1165359 filed and registered by theapplicant of the present invention discloses a method of extractingfeature points and generating feature point information that improvesthe above-described conventional method of extracting feature points andgenerating feature point data.

The feature point extraction unit 130 according to the present inventionextracts the feature points of a query image and then generates thefeature point information (feature data) of the feature points using amethod known by the conventional technology. Since the present inventionis not intended for a method of extracting feature points and generatingfeature point information itself and also may use a method of extractingfeature points and generating feature point information, which is knownby the conventional technology, without change, a detailed descriptionthereof is omitted.

The image matching unit 140 performs the function of determining amatching structure image ID by comparing the feature point informationof the feature points of a query image extracted by the feature pointextraction unit 130 with the feature point information of a structureimage stored in the reference meta-information database 160.

In this case, the reference meta-information database 160 is describedfirst. The reference meta-information database 160 stores the structureimage ID of a structure image of a structure on which an original imagehas been installed, the structure image feature point information of thestructure image, and additional information to be provided to the userterminal in association with the structure image ID.

That is, the reference meta-information database 160 storesmeta-information corresponding to the structure image ID, and allows theimage matching unit 140 to determine a structure image ID that matches aquery image.

FIG. 5 is a diagram illustrating an example of the configuration of thereference meta-information database 160.

Referring to FIG. 5, the reference meta-information database 160 storesan original image identifier (ID), a structure image ID, original imagefeature point information, structure image feature point information,structure location information, and additional information.

The original image ID is the ID of an original image installed(attached) on (to) a structure, such as a structure installed in aspecific space, for example, an outdoor billboard or the like.

The structure image ID refers to the ID of an image of a structure onwhich the original image has been installed, and the ID of the image ofthe structure in which the original image is included. For example, whenan advertising image has been installed on an outdoor billboard, astructure image is an image including the outdoor billboard on which theadvertising image has been installed, and an original image refers to animage of the advertising image itself.

The original image feature point information refers to the feature pointinformation of the feature points of the original image, and thestructure image feature point information refers to the feature pointinformation of the feature points of the structure image. In this case,the structure image feature point information is divided into featurepoint information inside an original image region and feature pointinformation outside the original image region and then stored.

The feature point information inside the original image region refers tothe feature point information of feature points present inside theoriginal image region, and the feature point information outside theoriginal image region refers to the feature point information of featurepoints present outside the original image region, other than inside theoriginal image region, in the structure image.

FIG. 6 is a diagram illustrating feature point information inside anoriginal image region and feature point information outside an originalimage region.

Referring to FIG. 6, original images are images of an “automobile,” theoriginal images are installed on structure 1 and structure 2,respectively, and structure image 1 and structure image 2 for therespective structures are indicated by dotted lines.

It can be seen that structure image 1 and structure image 2 are imagesof structures, including the respective original images, and include therespective original image, as indicated by the dotted lines.

In this case, the original images attached to structure image 1 andstructure image 2 may have different sizes or different left-to-rightratios, or may have, for example, different advertising phrase or thelike included in the images. The term “original image attached to astructure” used herein is based on a concept including the case where animage is completely identical to an original image and the case where animage is somewhat different in size, color or advertising phrase from anoriginal image. Accordingly, an original image attached to a structuremeans that the source of the corresponding image is an actual originalimage.

Meanwhile, in FIG. 6, reference symbol 601 denotes feature points insidethe original image region, and reference symbols 602, 603, and 604denote feature points outside the original image region.

In this case, the feature point information outside the original imageregion may be formed by feature point information that does not matchingthe original image feature point information when matching between theoriginal image feature point information and the structure image featurepoint information is performed. When matching between the original imagefeature point information and structure images 1 and 2 is performed, allfeature points included in the original image are matched, but featurepoints not included in the original image, i.e., feature points presentonly inside the structure image, are not matched. These feature pointsthat are not matched are considered to be feature points unique to eachof the structure images, and may be selected as the feature pointinformation outside the original image region.

The structure location information is the location information of aplace where a structure is located, may be implemented in the form of,for example, GPS information (latitude, longitude, altitude), and isstored in association with a structure image ID. Although GPSinformation does not vary with respect to the location information of astructure located in a fixed place, structure location informationcontinuously varies with respect to, for example, a structure imageinstalled inside or outside a bus, and thus the structure locationinformation is distinctively denoted with an indicator, such as“Moving,” which is indicative of a moving structure image, other thanfixed location information.

The additional information is stored in association with each structureimage ID, and is additional information to be provided to the userterminal 200 in accordance with a structure image ID determined to bematched. The additional information includes various types ofinformation, such as advertising moving image data, image data, web pagelink data, an audio guidance message, etc. The actual data of theadditional information may be stored in a separate additionalinformation database (not illustrated).

The image matching unit 140 determines a structure image, i.e., astructure image ID, which matches a query image by determining structureimage feature point information matching the feature point informationof the feature points of an query image, extracted by the feature pointextraction unit 130, by referring to the reference meta-informationdatabase 160.

The image matching unit 140 may determine a structure image ID using thefollowing method. First, the image matching unit 140 determines N (whereN is a natural number) structure image IDs of the highest matchingscores by comparing the feature point information of the query imageextracted by the feature point extraction unit 130 with the structureimage feature point information stored in the reference meta-informationdatabase 160. In this case, when performing comparison with the featurepoint information of the structure image, the image matching unit 140may compare only feature points inside the original image region, maycompare only feature points outside the original image region, or maycompare both the feature points inside the original image region and thefeature points outside the original image region, and uses anappropriate method as desired. These determined N structure image IDsbecome candidate structure image IDs.

Next, the image matching unit 140 finally determines a structure imageID by comparing the feature point information outside the original imageregion corresponding to the candidate structure image IDs with thefeature point information of the query image. That is, the structureimage can be accurately matched by comparing only the feature pointinformation outside the original image region, exclusive of the featurepoint information inside the original image region, with the featurepoint information of the query image.

After the above process, the image matching unit 140 may determine afinal matching structure image ID. Accordingly, structure image locationinformation and additional information stored in association with thecorresponding structure image ID may be determined.

FIG. 7 is a diagram illustrating the operation of the image matchingunit 140.

Referring to FIG. 7, reference symbol 701 denotes a query image selectedfrom among videos or images (captured images), acquired by the userterminal 200, by the query image determination unit 120, and referencesymbol 702 denotes an original image. Reference symbols 703 and 704denote candidate structure images that correspond to N (which is 2 inthis case) candidate structure image IDs determined by the imagematching unit 140.

As described above, the image matching unit 140 determines N candidatestructure image IDs by comparing the feature point information of thefeature points of a query image extracted by the feature pointextraction unit 130 with the structure image feature point informationstored in the reference meta-information database 160. In FIG. 7, onlythe candidate structure images 703 and 704 corresponding to twocandidate structure image IDs are illustrated for ease of description.

With respect to these candidate structure images 703 and 704, a finalstructure image ID is determined by comparing only the feature pointinformation outside the original image region with the query image, asdescribed above. Reference symbol 707 denotes the state in which thefeature point information outside the original image region matches oneof the candidate structure images, and reference symbols 705 and 706denote the state in which the feature point information inside theoriginal image region matches all the two candidate structure images.

Since the candidate structure images includes the original image, it isdifficult to determine a structure image that is accurately matched bythe query image using only the feature point information inside theoriginal image region. Accordingly, when the feature point informationoutside the original image region is used, information about thesurroundings of the original image region is included, and thus it ispossible to determine a structure image that corresponds to the video orimage acquired by the user terminal 200 based on the information aboutthe surroundings of the original image region.

First, the image matching unit 140 of the user terminal 200 determinestwo candidate structure image IDs, as illustrated in FIG. 7, throughprimary matching, as described above. The reason for this is that allsets of the feature points inside the original image match each other,as indicated by reference symbols 705 and 706.

Next, when secondary matching, i.e., matching between pieces of featurepoint information outside the original image region, is performed on thecandidate structure image IDs, only the structure image in the upperportion of the right side of FIG. 7 is matched, as indicated byreference symbol 707, and thus the image matching unit 140 may determinethat this image is a final structure image and determine a structureimage ID therefor.

Meanwhile, the image matching unit 140 may determine a structure imageID finally matching the query image from among the candidate structureID images using the location information, for example, GPS information,of the user terminal 200. For this purpose, the user terminal 200 alsotransmits the location information of the user terminal 200, acquired bya GPS sensor or the like provided in the user terminal 200, to the imageinput unit 110 while transmitting the video or image to the image inputunit 110, and the image matching unit 140 receives the locationinformation of the user terminal 200 via the image input unit 110 anddetermines a final structure image ID based on the location information.

Next, the additional information provision unit 150 is described withreference to FIG. 2 again.

The additional information provision unit 150 performs the function ofproviding additional information, stored in association with thestructure image ID determined by the image matching unit 140 using theabove-described process, to the user terminal side. As described above,the additional information includes various types of information, suchas advertising moving image data, image data, web page link data, audioguidance message, etc. Accordingly, for example, advertising movingimage data may be provided by acquiring the advertising moving imagedata from an additional information database (not illustrated) andtransmitting the acquired advertising moving image data to the userterminal 200.

Meanwhile, the additional information provision system 200 according tothe present invention direction may further include a calculation unit(not illustrated) configured to determine the directional angleinformation of the user terminal 200 directed toward the structureimage.

When a video or an image is input from the user terminal 200 and a queryimage is determined based on the input video or image, the directionalangle information of the user terminal 200 that viewed the structureimage at the time at which the query image was acquired (i.e., at thetime at which the user terminal 200 captured the structure image) may bedetermined based on the front surface of the original image that matchesthe query image. A method of calculating directional angle informationby comparing two images is widely known from the conventionaltechnology.

Using the method of the conventional technology, the directioncalculation unit (not illustrated) may acquire directional angleinformation based on the front surface of the structure image at thetime at which the user terminal 200 acquired the structure image. Whenthe directional angle information is known, a location where the userterminal 200 is placed may be determined based on the structure image,and additional information may be provided in accordance with thedetermined location. For example, the additional information of FIG. 5is classified as information for the left side and information for theright side based on the front surface of the structure image, and thenstored. Thereafter, whether a side in question is the left side or rightside may be determined based on the directional angle information of theuser terminal 200, and corresponding information may be provided. Forexample, when the user terminal 200 is located on the left side based onthe front surface of the structure image, information about shopslocated near a corresponding area may be provided, and thus moreaccurate customized information may be provided.

Since the directional angle based on the front surface of the structureimage may be determined at the time at which the user terminal 200acquired the structure image, as described above, not only the locationinformation of the user terminal 200 but also the movement directioninformation of the user terminal 200 may be determined using thedetermined directional angle.

That is, the additional information provision system 100 mayadditionally receive a video or an image of a specific structure imagewithin a specific time interval after receiving a video or an image ofthe same structure image from the user terminal 200, and may determine astructure image ID corresponding to a query image using theabove-described process. By doing so, the same structure image ID isdetermined with respect to the two input videos or images. With respectto the second input video or image, the directional angle information ofthe second input video or image, i.e., query image, based on the frontsurface of the structure image may be determined using the processdescribed in conjunction with the direction calculation unit (notillustrated). When this information is referred to as second directionalangle information and the directional angle information of the queryimage, acquired first, based on the front surface of the structure imageis referred to as first directional angle information, the direction ofmovement of the user terminal 200 may be determined by comparing thefirst directional angle information and the second directional angleinformation.

FIG. 8 is a diagram illustrating a process in which the directioncalculation unit determines the direction of movement of the userterminal 200.

Referring to FIG. 8, the user terminal 200 captures a structure imagefirst at the location of reference symbol 801, captures a structureimage second at the location of reference symbol 802 within a specifictime interval from the first capturing, and transmits the capturedstructure images to the additional information provision system 100.

The additional information provision system 100 determines a matchingstructure image using the above-described process, and also calculatesdirectional angle information at each location, i.e., first directionalangle information and second directional angle information.

In FIG. 8, θ₁ and θ₂ denote the first directional angle information andthe second directional angle information, respectively. When in FIG. 8,the value of the angle changes from 0 degrees to 90 degrees in the leftdirection and from 0 degrees to 90 degrees in the right direction basedon the front surface of the structure image, the direction of movementinformation of the user may be calculated based on the pieces ofdirectional angle information of the user terminal 200 of two queryimages. For example, the case where θ₁=θ₂ corresponds to the state inwhich the user terminal 200 remains stationary. Based on therelationship between θ₁ and θ₂, whether the user terminal 200 is movingto the left or right based on the front surface of the structure imagemay be easily determined. For example, when a movement from the locationof reference symbol 801 to the location of reference symbol 802 isperformed during a period from time t1 to time t2 (t1<t2), asillustrated in FIG. 8, θ₁ is an angle of about 15 degrees and θ₂ is anangle of about −15 degrees, and thus from this, it can be seen that theuser terminal 200 has moved from the left side to the right side.

When the direction of movement of the user terminal 200, i.e., thedirection of movement of the user, is determined as described above, itmay be possible to provide additional information while additionallyconsidering the direction of movement. For example, the additionalinformation of FIG. 5 may be classified for the directions of movementand then stored, and corresponding information may be provided to theuser terminal 200 in accordance with information about the direction ofmovement. For example, when the user terminal 200 is moving from theleft to the right based on the front surface of the structure image,information about shops located in the direction of movement can beprovided in accordance with the corresponding direction of movement, andthus accurate customized information provision is enabled.

FIG. 9 is a flowchart illustrating an embodiment of a method ofproviding additional information to the user terminal 200 using theadditional information provision system 100 of the present inventiondescribed with reference to FIGS. 1 to 8.

First, the user terminal 200 captures a structure image using a cameraor the like at step S100, and transmits a video or image of the capturedstructure image to the image input unit 110 of the additionalinformation provision system 100 at step S110.

The image input unit 110 transfers the received video or image to thequery image determination unit 120 and the query image determinationunit 120 determines a query image, i.e., a query target, using theabove-described process at step S120.

Once the query image has been determined, the feature point extractionunit 130 extracts the feature points of the determined query image atstep S130.

Thereafter, the image matching unit 140 determines a matching structureimage ID by comparing the feature point information of the featurepoints of the query image extracted by the feature point extraction unit130 with structure image feature point information stored in thereference meta-information database 160 at step S140.

Once the structure image ID has been determined, the additionalinformation provision unit 150 provides additional information, storedin association with the determined structure image ID, to the userterminal 200 at step S150

Although the configuration of the present invention has been describedwith reference to the preferred embodiments of the present inventionabove, it will be apparent that the present invention is not limited tothe embodiments but may be practiced in the form of various modificationand variations within the scope of the present invention.

For example, each component of the additional information provisionsystem 100 may be included in the configuration of the user terminal200, or may be implemented via a separate server. For example, the imageinput unit 110, the query image determination unit 120, the featurepoint extraction unit 130, and the image matching unit 140 may beincluded in the user terminal 200, and only the additional informationprovision unit 150 may be implemented using a separate system. In thiscase, the components of the user terminal 200 may be implemented in theform of, for example, applications.

Furthermore, the image input unit 110 and the query image determinationunit 120 may be implemented in the user terminal 200, and othercomponents may be implemented using a separate system.

Aside from these methods, it should be noted that componentsconstituting the additional information provision system 100 may beconfigured in various separate manners in terms of the load problem of anetwork, user convenience, efficiency, etc.

1-11. (canceled)
 12. A server, comprising: a reference meta-informationdatabase configured to store at least a structure image identifier of astructure image of a structure on which an original image has beeninstalled, structure image feature point information of the structureimage, and additional information associated with the structure imageID, and wherein the server configured to perform functions comprising:receiving a query image; extracting one or more feature points from thequery image; generating feature point information based on the one ormore feature points; determining a matching structure image ID bycomparing the feature point information with the structure image featurepoint information stored in the reference meta-information database;providing additional information associated with the matching structureimage ID.
 13. The system of claim 12, wherein the query image is relatedto an input video, and wherein receiving the query image comprises:receiving the input video; extracting a plurality of frame images fromthe input video; and determining the query image from the plurality offrame images.
 14. The system of claim 13, wherein determining the queryimage from the plurality of frame images comprises determining the queryimage based on one or more gradient values of the plurality of frameimages.
 15. The system of claim 14, wherein determining the query imagebased on the one or more gradient values of the plurality of frameimages comprises determining one or more scores based on a scorefunction operating on the one or more gradient values.
 16. The system ofclaim 12, wherein the structure image feature point informationcomprises feature point information inside an original image region andfeature point information outside the original image region.
 17. Thesystem of claim 16, wherein the feature point information outside theoriginal image region is formed by feature point information that doesnot match the original image feature point information.
 18. The systemof claim 17, wherein the functions further comprise: determining one ormore highest structure image IDs of highest matching scores by at leastdetermining a structure image ID of the one or more highest structureimage IDs based on a comparison of the feature point information outsidethe original image region corresponding to a highest structure image IDof the one or more highest structure image IDs with the feature pointinformation of the query image.
 19. The system of claim 18, whereinreceiving the query image comprises receiving location information withthe query image, and wherein the reference meta-information databasefurther stores at least a structure location related to the originalimage.
 20. The system of claim 12, wherein the functions furthercomprise: determining first directional angle information of a userterminal directed toward the structure image.
 21. The system of claim20, wherein the functions further comprise: receiving one or more secondimages; determining a second query image from the one or more secondimages; extracting one or more second feature points from the secondquery image; generating second feature point information based on theone or more second features points; determining that the matchingstructure image ID is associated with the second image based on acomparison of the second feature point information with the structureimage feature-point information; determining second directional angleinformation of the user terminal based on the one or more second images;and determining a direction of movement of the user terminal based onthe first directional angle information and the second directional angleinformation.
 22. A method, comprising: receiving a query image;extracting one or more feature points from the query image; storing atleast a structure image identifier of a structure image of a structureon which an original image has been installed, structure image featurepoint information of the structure image, and additional informationassociated with the structure image ID; generating feature pointinformation based on the one or more feature points; determining amatching structure image ID by comparing the feature point informationwith the structure image feature point information; and providing theadditional information associated with the matching structure image ID.23. The method of claim 22, wherein the query image is related to aninput video, and wherein receiving the query image comprises: receivingthe input video; extracting a plurality of frame images from the inputvideo; and determining the query image from the plurality of frameimages.
 24. The method of claim 23, where receiving the query imagecomprises determining the query image based on one or more gradientvalues of the plurality of frame images.
 25. The method of claim 24,wherein determining the query image based on the one or more gradientvalues of the plurality of frame images comprises determining one ormore scores based on a score function operating on the one or moregradient values.
 26. The method of claim 22, wherein the structure imagefeature point information comprises feature point information inside anoriginal image region and feature point information outside the originalimage region.
 27. The method of claim 26, wherein the feature pointinformation outside the original image region is formed by feature pointinformation that does not match the original image feature pointinformation.
 28. The method of claim 26, further comprising: determiningone or more highest structure image IDs of highest matching scores by atleast determining a structure image ID of the one or more structureimage IDs based on a comparison of the feature point information outsidethe original image region corresponding to a highest structure image IDof the one or more highest structure image IDs with the feature pointinformation of the query image.
 29. The method of claim 28, whereinreceiving the query image comprises receiving location information withthe query image, and wherein the method further comprises receivinglocation information with the one or more images; and storing at least astructure location related to the original image.
 30. The method ofclaim 22, further comprising: determining first angle information of auser terminal directed toward the structure image.
 31. The method ofclaim 30, further comprising: receiving one or more second images;determining a second query image from the one or more second images;extracting one or more second feature points from the second queryimage; generating second feature point information based on the one ormore second features points; determining that the matching structure IDis associated with the second image based on a comparison of the secondfeature point information with the structure image feature-pointinformation; determining second directional angle information of theuser terminal based on the one or more second images; and determining adirection of movement of the user terminal based on the firstdirectional angle information and the second directional angleinformation.